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Saul I Takalo 



1 Ob|actlvoa • tttchlnoj - tvtluttlont tho r?o*ltfi» •! 

match 

Oit of tht blggaat problom In ■•■•{•■f 1 

tctlvlty It how to natch objoctlvoo, tooofThw. ^ •♦•JJf*^ 

ion. ll oihor wordo. how tho Mnlond^^J^ 

trontformtd Into Iht -Implomontod eurr Icul**" f 

Into tht "rtall«d curriculum" a no tlwplo mottor • Toot- 

book wtltort tnd toochoro ,nt * r *"* 

curricula tnd • lfitilomont« It to o cortoln 

covtr tomt polntt thoroughly, tm In o J***' 

do not tttch tomt polntt tt oil, tnd *y°*" rttt •••ft* 

oomt conttntt not montlonod In tho I"**"** currlwiltwu 

Studontt, in turn, loom <"'••»«•"> '^.JS!*^!? 

curriculum to o cortoln o*tont. Tho llnho 

thrtt concoptt trt lliuttrttttf In tho following fifyro. 
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CURRICULUM 
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CURRICULUM 
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Tho broken llntt trt Inttndtd to USuttrttt thtt thtrt oro 
no almplt wtyt of gtttlng from ont otogo In tho 
onothor. Thtrt It t lot of Intorprotot Ion ond thlt moont 
thot thiro lo o nood to dovolop oxpllolt ond rlgorout 
procedural rultt to mokt tuch Inttrprtttt Ion loot od hoc. 

Ttttt tnd txomlnotlont trt dltt Ingulthtd from otol- 
uotlon. Thlt It mttnt to convoy tho Idoo thot ovoluot on 
lo o broodtr (tuptrordlnttt) ttrm thtn ttttlng. It It tlto 
moont to auggaat thtt ttttt tnd a*amlnatlona oro not 
nocoooory for making avtluet lona. EvoluoMon moont making 
ludgtmtntt tbour tht worth (merit, voluo) of emthlng. In 
•II welke of life wt art conatantly making ouch 
mtnto. In tducetlon, owoluttlon ututlly meene c «^«f> n « 
what haa baan loornod to what woo Intondod to bo tauofit. 
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22 xirs^r^ 2^'?2.r.2 M 'T n '? l -»-. ^ 
on°. n, ;; b ^°:.;: r: • -'-ion.?';:.,; 

"-king evelu.tion;: ' ,,,M "« d '" <'nfcrm.t Ion) for 

2 Criterion- .nd Norm-Ref erer.ced Me..ureo*nt 

1970'.. Pr.ctlc.lly Tf thLT^ tow.rd. the end of the 

deed.. Yet, crit ic ,. ' „ -''' P«bli.hed during thai 
• new idee. ' 'erenced mee.uremenf i, not , U ch 

•b.oiui. .nd 2°.u2 ™:.2L2 ou ! ,he di "—— 

Around 1950 ». , "Vi J™ ""enty X-r. .go. 

•tud.e. on .b.olu?. .nd „ ^ 'J'"" 1 ° u ' 
theoretic. I b..i«. S " gr.ding , n d on their 

we. introduced. The Ide. IIT f cr 1 1 " 1 ° n - re f erenced te.t" 
not le .d to fir'slr wo" H J" • ' " , ' Ved bu ' »« 
Hj.ek took up the conM»r Ind 2 • 1 ? < ' """" Poph,m ' nd 
Infliction.? COncept ,nd expl.c.ted further .ome of 



-v^nH-T? M2:::'3.irt.,: h : r^oT* 0 '' 1 ° bi - c "- 

•mergence of cr ii.r 0 n-2f ,2nrPd """" ln ,he 

outlined te.ching program, w M «"«r<w»m . C.refully 
ol.tr ibur ion of 9 JcoreT^i, IL ,Q ' n ° f ""' 

effective. There .hou 5 be . hioh" 09 "™ ' ndeed - 
•rore. .nd . decre.te in !.r! 9 , P ercenl "°e of high 
•tic for cl...i„ ";, The '-"er i. problern- 
°<ce. rely he.vi y o v.H.nc'e ?h«",T OJl °' 

criterion-referr. ed , e.t ? .7 i • ' V ' ' ' eb 1 y '" bel 
New eppro.ches were c » * ^ I 9 ,° 1 ' ow f e I i «b i 1 i r y , 

J969). fe Cre "'y needed (Poph.n , n d Hu.ek, 

le.rnfng'ind iS^T'Si 722,2 re '" ed l ° 

done to di. cover 1 "« progr.m., i, , he wor k 

(t-.k) hler.rchie. ".g^ 9 e V Ml . n"" 

Thi, ,,ork revealed rhor The 1'"' Resn '<*. 1 967). 

require, . through .n.fyi?! nf'.h '""log outcome. 

Pre.in,,„. ry stPp I o V e^tX ue , i 2? ,n8 " er " ' 

Mr.. P.P^r. P. 18 " Pr ° P0S ' M OI.2r'J 

•« M -..r,i Uy Poph«: , (,;7 , 0f , ~ ,l 9 ;?? C :» d « ' ' " ' " °" h. , oeen 
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reepect to • well-defined btthovlorol domain.* Thfte meene 
that the Interpreteblllty of iho teet reeult It of primary 
concern. Whereat In >iornweff oroneod moOeuroMnt on 
Individual's toat acora derloee Ite mooning mo Inly from 
Ita ralallonahlp to tht ceoroc of othor tx ami no a a (relet- 
Iva Interpretetlon), tht acoroa on a crlter len-refferencool 
taat darlva thalr moaning from tht •cores' relet lonahlp to 
a claaa or domain of teoke (tbttlutt tnterpretetlen). Twit 
a domain acora can ba Interpreted In ttrma eff what en 
Individual can do ond what ha cannot do im II alee Irtdlc- 
ataa what proportion of all poeelble taakt (Itema) of tho 
wholo I am onlvaraa tha Individual could Moo eeltod If 
they wo> a eOiln letered to him rothor thon only o eomple off 
thorn. A domain acoro larda Itaalf ta obttlott Interpret st- 
lona and can ba uaad both for guolltotlvo ond ojuont ttot lee 
datcrlptlont (what It motttrod ond how much It moetered). 

Savaral terme for thla kind of toatlnf novo batn pro- 
poaad within tha cr Itar Ion- referenced movomont • Cool 
(1962) propoaad a tarm "content -at ond-trd I icol" to deocrtbe 
a taat which praducaa taat acoroa which Indicate what po»- 
centege of a ayatomotlc eomple of daflnod toaka o P"*** 
haa aolvad corroetly. Ooborn (IMS) ueod tho torn m±- 
vtrtt-dtf Intd teat" to rafar ta a taat which praducaa on 
unbTese3 "oaf Tmafa of hla acoro In on explicitly doff I nod 
Itom content unlveree. Hlvaly (1973) preffere tho term 
"domain-referenced taot" oa m laaa ewoltloue tarm thon 
unTvSrt^3a?T^a'foatrXorvar (1974) haa advocotod tha uaa 
of adumttrlc (rathar than traditional paychomotr Ic) taata 
to maaaura wl thin- Individual growth (compatanca) Inataad 
of batwean-lndlvltfuel dlffarancoa (ability* Intol I Igonco) • 

Tha tarm "ob) act I vaa-btttd taat" haa aomot Imoa boon 
ueed aa a near -synonym for cri tar Ion-raff artnetd taata* Iff 
the Item* art almply dar Ivad from bohovloral objoctlvee 
without a atrlctly predetermined procedure* however* 
object Ivt-btttd teete do not lend thomatlvtt to criterion- 
referenced Int erpr etet Ion. 

The term "maetery teet" hee been dor I ved molnly from 
the maetery leernlng eyetem developed by dloom (196S* 
1971), lergely on the beela off tho model off echool leern- 
lng propoeed by Cerroll (1963). The meln purpoee off 
mattery teete le to help In the cleealff lent Ion off ttudtntt 
ta matttrt or norvneetere off tn objective In order to 
fecllltete the mentgement off an individual I ltd toechlng 
program. 

If one wore ahown a taet which only contelned the 
Instruction* to etudente and the teet Items, It would be 
difficult to eey whether tht tett It t crl ter lon-reffer- ^ 
encad teat or e norm-referenced teet. In order to bo eble 
to make that decision It la necetaery to know how tho_toot 
wet produced. It It in the work prior to tht eeetmbly off t 
Test tfitT'moit of the effort needt to be epent In produc- 
ing e criter <*n-referenced tett. Differences between two 

f r BEST COPY AVAILABLE 
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formt of cr iter ion-ref erenced lasting (domeln-ref erenced 
and mattery tasts) and norm-ref arancad tatting ara surrmer- 
Ized In Tabla 1. Tha first flva stages In tha davatopmant 
of teste rafar to tha planning staga and th* rast to tha 
tachnlcal espects of tatta and their uses. 



3 Stages In tatt Construction 

3.1 Specification of Content 

It It In the specif Icet Ion of lhe^cont_enl_dqrnajji that 
the greatest cheTl enge "~sn3 "eTso * "fhe'gree fe st merit of 
crlter ion-ref erenced testing lies. In treditionei norm- 
referenced tests the content limits' ere only oertielly 
specified. Short i ns t r uc t i one I end beheviorel objectives 
ere used es the besie for i t em generet I on . As Bormuth 
(1970) end Anderson (1972), amonu, others, heve ehown, 
there Is so much room left for interpretation that the 
items may reflect the cherecter ist ics of the test con- 
structor more then those of the instruct ionel program. Too 
much room is left for crestlvity, which eccording to Pop- 
hem (1978, 1980), is not es desireble es strict edherence 
to the content limits. Severe! methods heve been proposed 
for making domain specification more edequete. These will 
be discussed below in some deteil, since this is e crucial 
part of sll cr I ter I on-referenced meesurement . 



ii?!? JL 1 5DI 1 0 rm * £ f. 20i 

Bormuth (1970) hss suggested that linguistic snelysis 
besed on t rensformet ionel grerrmer could be used to make 
explicit the methods by which items ere derived from 
stetetnents of instructions! objectives. Oormuth sdvocetes 
operet i onel i sm es e wey of Introducing rigor Into item 
construction end sees syntectlc operet ions es e promising 
wey to do this. His method is illustreted below. It sSows 
some item t r ens f ormet i ons that heve bren performed on e 
sentence "The older sister put out the fire." Using syn- 
tectic trensf ormet ions severel comprehension questions 
could be esked ebout the sentence. 

Trensf ormet i on Name Question 



Ccho The older sister put out 'he 



fire? 



Teg 



The older sister put out the 
fire, didn't she? 



Yes -No 



DM the older sister put out 
Hi* fire' 
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-IS" 



Noun do I at ion 



Wo put Mit th« flrot 
Miot did IM elder oil 



ilicer out 

out? 



Noun modifier deletion WllcH ildtdr p« •« «»• 

i 

It ..em. obeloue thot BoMoith'o Mlktd lo • Jf" 1 "" 
tool "or "%.r.tln, Item, feting the ••^'f***;"^; 
written ond epoken dlocouroo. Andoroon <" 7 " K"i ,! 
•erne othor e.emple. «i » — g?!* '^Ll! 

toot dlocouroo comprohonolon. On* weeknoee 
mothodo lo, howeeer, thot the onpficolo lo on 
level oporotlono rothor thon dlocouroo lee.l unite. Recent 
Stl ml'* ecouro. en.ly.l. by Hell Hoy end He..n. »•» 
Dljk, hwyer ond othoto will be of ooo In mooing from 
•ontonco to dlocouroo-lovol tooting. 

M^£lnj_Sontonco 

Mopping oontoncoo oro uood In fnoet ooolytlt 
od by Outline. U»o»>. Fecet enelyele oon *• J" 

ocrlbe tho bjunderlee ond otruciuro of o dam In of tooting 

condition.. Feet, or. <*••• ?' eh 'T.«? 
on which Item. In o gl»en dumeln con ^lUt. Feee enjl- 
yolo woo uood by tho prooont writer In 1MO In on •"•"{" 
to concoptuollto tho dome In of ^'^IT^mI! ThJ ?ir.7 
IEA Internetlonol Study of Wr tton Cowpooltlon. Km firet 
ottempt lo Illustrated bolow. (For o loto? eerelnn, eee 
Tokolo, 1962.) 

Mlllmon U»7e> oloo uood focot enelyele In hie etudy 
of how tho foim ond contont of Itomo oro reletou to Itom 

difficulty. 

A mplifi ed Objoctlvoo 



Aftor finding out thot Itom gonorot Ion on tho beele 
of trodltlonol bohoilorol objective woo •<*!•«* «• }"° 
much Intorprototlon ond thot uolng » tOT W J!.;2? 




Ing oxwiplo llluotrotoo amplified objoctivoo 
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TABLE 1. Characteristics of Two Types of Criterion-Referenced Tests and of 
term-Referenced Tests (adapted fro* Mlllman, 1974, and Berk, I960). 



Stages of 
Development 



Alternative Conceptualizations 



Criterion-Referenced Testing 
Don 1 n -Referenced Hi s tery 



Norn-Referenced 
Testing 



1. Specification 
Of Content 
i1n 



2. I tea Con- 
struction 



MaxlM specification 
of content Units 

Methods: 

1. Item transforma- 
tions 

Z. Mapping sentences 

3. Algorithms 

4. Item forms 

5. Amplified objec- 
tives 

6. Test specifica- 
tions 

Generation rules 



3. Specification Infinite or finite 
of Item Item universe 

i1n 



Content limits only 
partially specified 

Methods : 

Instructional and 
behavioral objec- 
tives 



Traditional rules 
Infinite ? 



4. Item Analysis Purpose to detect 

flawed Items 

Methods : 

1. A priori judge- 
ment o f Item- 
objective r on- 
gruence by suu 
ject matter 
experts 

2. A posteorlorl com- 
putation of Item 
statistics 

5. Item Selection Random 
from Itc*m 

Universe 



Purpose to detect 
flawed Hems 

Methods: 

? 



Nonrandom (?) 



Content limits only 
partlcally specified 

Methods : 

Instructional and 
behavioral objec- 
tives 



Traditional rules 
Infinite ? 



Purpose to select 
Items 

Methods : 

A posteriori compu- 
tation of Item 
statistics 



Nonrandom 
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TAW.E 1 (cont.). 



Stages of 
Development 



A1 tirnitl vi Conceptualisations 



Crlterlon-Neferenced Testing 
Ooma In-Referenced Aesttry 



Ytttlnf 



6. Cut-off Scort 
Selection 

7. Validity 



8. Reliability 



Optional 

Content 

Construct 

Decision 



Required 

Content 

Criterion-related 

Construct 
Decision 



Consistency of 



decisions 



9. Score Inter- 
pretation 



10. I ten and Test 
Variance 



1) Consistency of 
decisions 

W 0 . t) 

2) Dependability 

(♦(*)) 

3) Error of ftaasmre- 
ment or estimate 
around dona In 
score using ♦ or 
other Indices 



Performance In rela- Performance In rela- 
tion to domain (level tlon to required 
of functioning) level of mastery 



ftmmlred (?) 

CrlWlon-r\*lated 



Tred1t1omal w 
cemures (beted 
correlation) 



Perfonaance In rela- 
tion to required 
level of mastery 

Not required 



Not required 



Performance 1n rein 
tlon to other 
examinee* 



Required 
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topping Scnttnct for tht Domain of Writing 
Following Cuttaan's Facet Analysis Scheme 



A. Activity 

1. Receive 

2. Send 



a/an 



B. Channel 

1 . auditive 

2. visual 



nessage which 
deals with 



C. Content/topic 

1. self 

2. school 

3. hoce town 

4. hobbles 

5. 
6. 



and whose 



D. Communication Partner 

1. addressor 

2. addressee 



£. Role relationship between 
addressor and addressee 

1. a higher social status 

has/is 2 - an e( * ua1 5X141 $utus m«a < 
na$/1$ 3. a lower social status and * ich 1s 
4. identical with addressor 



F. Degree of publicity / 
formality 

1 . private 

2. serl -public 

3. public 



consisting of 



Input-output relationship 
[ stimulus -response ) 

1 . repetition cf Input 

2. modification of Input 

3. internal Input 



and whose 
purpose Is 



H. Function 

1. to preserve the message (documentatlve) 

2. to Inform (referential) 

3. to persuade (emotive) 

4. to describe (descriptive) 

6.' 



Different configurations of variables lead to different rhetorical modes (narrative, exposition, argumentation, etc.) 



Examples : 

A2 ♦ B2 ♦ C2 ♦ D2 ♦ £1 ♦ F3 ♦ G2 ♦ HI - a personal letter to a friend 
A2 ♦ B2 ♦ C2 ♦ D1 ♦ £3 ♦ F2 ♦ 64 ♦ H2 . letter of application 
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Ob|ec t I ve i Given • santmce with • noun ot Vorb emitted, 
* the student will eelect ft am two eltornotlvoe 

tht word which most specifically or concretely 

completes lha sentence. 

Sample Item 

DlrtcUontt Mirk an "X" through ono of tha worda In 
peranthaaai which makee tha aantanoa do- 
■crlba a claarar picture. 

Exempt at Tha racar (tumbled, want) down the hill. 
MjetlUad JObjo^ tLa 

lililM-Sim i * on 

1. Tha student will ba glvan almpla aantancaa with 
tha noun or varb omitted and will ba esked to mark on 
"X" through tha ona word of a glvan pair of alternat- 
Iva words which mora specifically or concretely com* 
plates tha sentence. 

2. Eech test will omit nouns and verba In approx- 
imately aqual numbers. 

3. Vocabulary will be familiar to e third or 
fourth-grede pupil* 

Reigqnie_AUernet i vei 

1. The ttudant will be given pelri of noum or 
pelre of verbe with distinctly verled degraee of de- 
ter Ipt I ve power ■ 

2. In pelre of verbs, one verb will either be e 
linking verb or en ectlon verb descriptive of generel 
action (e.g., la, goes), end one verb will be en ect- 
lon verb descriptive of the menner of movement In- 
volved (e.g., scrambled, skipped). 

3. In pelrs of nouns, one noun will be ebetrect or 
vague (e.g., man, thing), end one noun will be con- 
crete or specific (e.g., cerpenter, computer). 

Cr l^i Hon _qf _Cqr rectjness, 

The correct enswer will be en "X" marked through 
the more concrete, specific noun or through the more 
descriptive ectlon verb in esch given pair. 

(Sources Ml I linen, 1974) 

While amplified objectives clesrly define the mea- 
sured domeln snd specify item generstion in greater detail 
thsn simple beheviorsl objectives, Pophem (1980) observes 

ii 
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that this et tempt to "shoot for f u.t the right balance be- 
tween c I 5 * - — ' — - - 
much re 
writers, 



tween clarity end cone I senrss " failed. There was st I I I too 
much room left for the per.on.l i nt erp. et?t i on of i tSm 



Experience with amplified objectives | e . Pophem vid 
Vil.ilZ llei9U " . 10 Relieve that . so-ceMed limited focu. 
strategy wet desirable. This means that the strategy is to 
focus measurement and to limit it to "a smeller number of 
!! , fl , f 1V,0r,f bul t0 conceptueliie these beheviars 

so that they W ere large scale, important behaviors that 
subsumed lesser, en route behaviors" (Popham, 1980, p. 

The test specification consists of t) a short general 
description, and 2) a sample item, which give the reader . 
genera! Idea of what the test might contain. These .re 
1? !Tf X , L ' df ' al, * d -Pacification of the stimulus 
attribute! and 4) response attributes including specific- 
ation of the correct answer and, in the case of multiple 
choice .terns, of the reasons for .arious distractors. The 
test specification is illustrated below (Tekala, 1984). 



Doma»n Specif i.cetiqn > .end Hem Genera t I on Rules for 
Vocabu [an Size Assessment" 



Be h a v i or 

(1) When given <> Finnish word in writing, the student 
can produce an acceptable English equivalent in writing 
recall or active vocabulary). (2) When gi/en an English 
word in writing, the student can produce an acceptable 
vocSbul.r?)! V nt 10 WriMn9 ( '« 0 * ni "°" or passive 
St ,jt*j hji spec i f i can on 

The vocabulary presented in the core te^ts and extra 
optional) texts in widely u , e d English textbooks is 
listed. A stratified random sample is selected from the 
universe of such word lists. The words are presented with- 
out providing any context. Some of the words are used to 
measure both the passive and active knowledge of word 
meanings. y wui u 



Response spec |f i cat i on 

The s t uilfnt has t 
provided for ih«r purpii^e. 



The student has to write the response in t fie space 
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Scqrwjs 

Th. r.»pon.oe oro eeored 0 - 1. A 
•ccoptiblo md und.r.tendeblo >*lch inoy 0 

InSnioa .troti, !■ icotr" 1. In eeorlng octl»o ••eobul- 
.Ty, diction I. be.od on how IM wrltt.n EnglUh word 
m' d .ound If rood .loud. Thut, th. .tud.M will got •-•« 
mtrk. If he/.»-e ho. given th. Engl I. h oqolvelwt „•» 
H'nJ ' wor "t.lo* •> "h.u.- In.t.ed of "*•«•■•" 
"heu* In Flnnl.h orthogrephy -ort.ipondi to tno way 
"hou.a" I. pronounced In Englleh. 

S ornp 1 . I t WW. 

Initructlonn "In thli toil you can ihow how wall JOU 
know tho Englleh »ocebulory Includ.d In your cour.o work. 
BouL t. pr.e.nt.d . n«nb.r of Flnn.sh word.. >out to.k 
ia to writ. th. Englleh .qulv.l.nt on th. I In. obove tho 
Flnnl.h word. Writ. th. word even If you my not b. JOlM 
■ur. .bout th. corr.ct .palling, .Inc. epei I Ing ml.t.k.. 
■ re a minor conaid.rat Ion In ■coring.' 1 




"Writ. .n. Flnnl.h aqulv.lanta of th. following Englleh 

words." 




Pophen, (I960, 1981) fool, th.t ta.t .p.clfle.tlon. 
Ilk. tha ooa .hown In th. .bov. constitute • '••""•""J 
b.l.nc. b.tw..n cl.rlty and "ncleeneee eo thet buey 
people Ilk. toachora might not b. put off by •"'• n * 
apeci'lcity. Teet .p.clfle.tlon. c.n alio contain a 
eupplemont, which c.n glva additional guldens In how to 
select etlmull, how to phr..e que. t Ion. , end .0 on. 



3.2 Site of Domain 

Th. proper elie of th. dome«n o many 

in te.tlng, ultlmetely dependent on the purpoee of the 
ta.t (me.eurement). A f.lrly l.rg. domeln I. W'* 9 ' \V.l 
if we ere Intereeted In more g.nerel forme of "termlnel 
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behaviors" (i.e., we are doing "aurrmat I ve" evaluatior and 
giving gradea). A more limited domain definition it re- 
cortmended when we are more intereated in M en-route H be- 
haviort end need information for deciding whether we need 
to review aome matters with all atudenta or give remedial 
help to aome atudenta ("formative" and "diagnostic" 
evaluation). Let ua illustrate the issue of domain size 
with aome concrete exempiea. Compare the following domain 
de f i n i t i ona . 

1. Student can apeak English 

2. r *„uent can ask for information in English 

3. Student can ask about (.) time In rnqlish 

(b) place 

(c) cost 

(d) another person's feelings 

( e ) anot her person's pre f er encea 

(f) another person's opinions 

(g) another person's advice 
etc . , etc. 

4. Student can ask for 

Ca) conformation of information (i.e., make yes/no 
ques t I ona ) 

(b) lacking information (make questions with *OW. 
WHAT, V+CN, etc.) 

5. Student can make questions with WHAT 

6. Student knows what WHAT means 

7. Student knows how to apel I /pronounce WHAT 

Points 1 and 2 are probably too broad dome • n definit- 
ions to be of much uae to language teachers. Levels 3 and 
4 are probably cf appropriate size for surrmafive evaluat- 
ion. Levela 5 through 7 might be considered for diagnostic 
I urposea . * . 

Tollov.ing Baker (1974) and Pophan (1975), it ahould 
be emphasised that we ahould not test trivial matters be- 
c*use that may lead to excessive testing. We should only 
teat important matters and reserve as much time as poss- 
ib e for teaching and learning. Our testt ahould include 
only auch items that contain those featurea and elementf 

7^?!? V,H fni^ mak " 8 d,,ference in ""dent response 
IMillmen 1974). To take a simple example, it probably 
makes no difference whether we use "he" or "she" to test 
whether students car use the a-form in the aimple present 
*ns*. If this is so, it is superfluous to have two items, 
one with "h •" and one with "she". 

Trying to concretize further rhe issue o ' oma i n 

size, tt is probably not useful to treat as a d .»ne- 

thing that can be t au ,* id l^rned in jne ie the 

other hanl, if something takes a whole term I fhat 
probably bp it Jtvidett into more than 0 do, 
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3.3 Levels of Mtiiuremtnt 

Knowltdqo ond skill ore not dlctiotoftwut pfienomone, 
l.o.» It lo seldom the coto thot we either know ond con tfo 
something or do not know or cofinot do It. ThOro iro 
vtrlous dogroeo of knowledge and vorleuo level 6 of tklll. 
Know I id gt con bo portlol or cemplett, ond •kilt con rtnfo 
from thot of o novice to thot of mot tot (export). One of 
tho mo it Import ont polnto to keep In mind In oil mfcootfrt* 
mtnt ond tutting lo thot It thou Id itfiiltlveiy porlro> 
■uch o rongo of knowlodgo ond tklllt. wo do e tfloentvlee 
to tttchtn ond ttudtnti, ond undermine tho frelO of thO 
ichool, If wt mtoiuro only ot higher level! on* thvo 
undoroit Imeto tho of feet Uoneoo of toothing ind looming. 
Thli principle con bo llluitrotod by tho following figure* 



Level of 

kmwltdot/ 

skills 























V///M 



Itow/Ttsfcs 



Tho outhor hoi dltcuntd thli quo it Ion In greotor 
dotoll In hlo dlitortotlon which detlt with vecebulery 
learning. Due to ipoco llmltotlom, on Intoroitod roodor 
la roforrod to thot publlcotlon (Tikilr>, 1984, 63- 
67 } 04-65). Soo ilio Appendix 1. 



4 Construction end Selection of Item* 

In the construction of Items certoln genorel rules 
hsve boon devised for producing trodltlonol norm-refer- 
snced tests. Such edvlco Is presented In e nunbor of books 
which doel with testing end ovoluotlon. Most of thoeo 
rules ore olso oppllceblo to crlterlon-ref ertncod mooiuro- 
mtnt. The only difference Is thot more stringent domendo 
ore sot for tho procedure In Item gonerotlon. It It, for 
Instonce, very Impart en t to stick to tho lluuts oet for 
tho st Imulus ond response choree tor 1st lea. Convergent 
rethor <hsn divergent creel Ivlty Is needed In Item Gener- 
ation. Work corrltd out by Corroll (1968, 1976) Is of 
Interest In this respect even if It Is not In tho moln- 
streom of cr I ter Ion- rsf ertncod meoiuroment • Rold snd Hole- 
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dyna (I960; 1982) alio provide a useful review of recah. 
advances In the Item-writing technology, Including 
computer-based methods (cf. ilio M!i!,nin I960). They note 
that the major positive result of the Increased attention 
to the process of Item writing it the heightened concern 
for the loglcel congruence between Instruction end teat- 
I ng. 

Once the rulea for domain definition end for Item 
generetion have been worked out, It it neceaaary to con* 
■idar epeclflc Items* Unlike In norm-referenced teating, 
It la necemry In cr 1 1 er i on- re f erenced teating to knov, 
what the univerae of Iteme la that represents the defined 
domain content* This univerie can be finite o? infinite* 
AaMIIJmen (1973) point! out, It ii not neceasaiy that the 
population of Items actually exists. Wr.at is necessary, 
though, is that the domain is to well described that a 
high agreement can be reached about whst • I ttm are and 
what are not members of the population. 

Further, unlike In norm-referenced and mastery teata, 
it is necessary to drew a r an dom a amgje from the univerae 
of ail possible iteme because TonTy" "f h i s procedure makes it 
possible to produce an estimate of the ex.v.inees' total 
domain scores. Random sampling of items is needed in order 
to meke ir possible to generalize into the whole domain 
tested. It is generally assumed rhat 10-20 items ere 
needed to measure a gi/en content domain* 



5 Validity as an Issue in Criterion-Referenced Measurement 

Cr i r er i on- re f en need tests are more end more often 
used i n won i for I ng 'ndividuel progress through objectives- 
based instruction! programs (formative testing), to dia- 
gnose learning problems (diagnostic testing), to evaluate 
educational and social programs (program evaluation), and 
to assess level of performance on certification and 
licensing examinations. The usefulness of such applicat- 
ions depends heavily on the validity of the procedures 
undertaken in such testing. 

According to Hambleron (1980) validity cona i de r at I ona 
in cr i t er i on- ref erenced testing arise at thee steps: 1) 
the selection of objectives (content domain;, 2) the 
meaaurement of objectives (content domain) included in the 
criterion-referenced te»., and 3) the uses of teat acorea. 

Validity is a difficult topic in all meaaurement and 
c r i t er i on - re f erenced measurement ia no exception. Term- 
inoloyy varies quite a lot so that different terms are 
usea to designate the same cha r ac t er I s t i r „ I *ne same 
term is uss'J to designate somewhat different thing. There 
irt? also soir>e f un J amen f a I confusions that have persisted 
for a I ong t ime . 
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Aa Cronboch (lftl), Muoololt <1?7* ) oM tit* CIf*f) 
novo pointed out, • mojor concoptuol oonfuolon or 1 00*8 from 
the f.ct thot coot tot validity !• fooueod on toot formo 
r.th.r than tool ocoroo, on Inetruwofite ***** 
rneoouromonto. In Llnn'e wordo •quoetlOlle of J*»f}«J 
quootlono for tho eoundneet if tho Inter^rotOtlOll 0? o 
meeeuro ... Thuo, U lo tho Intororototlon rothor tho* tho 

mo.our. Choi lo v.l % J*V£\l ttl^If 
mony Intorprotot lono which dlffor ** V*[*^*T*°[ 
volldlty ond In tho typo of 0*1 donOO rOJ^If^f Of 
volldotlon procooo" (Linn, lf7f, p. Mf). "i*** W T^H 9 
Moooick ototoo thot contont coverage lo on 
oidorotlon In toot construction -*d IntorprototlOfl 0U% « 
dooo not itoolf provldo volldlty. Ho would prefer tho torn 
"contont rolovonco* or "contont roprooontotlvonooo", olnco 
thoy do not roolly provldo ovldonco for tno volldlty of 
tho Intorprotot Ion of ocoroo. 

Pophom (1978) uooo tho torm •domoln-ooloctlo* Villd- 
Ity* to rofor to tho ^ueetlon of how wo 1 1 tho roou l tt ob- 
tolnod con bo generellied to oo mony othor " 
poooiblo. It thuo roftombloo "conotruct volldlty" to oojm 
extent, elthough tho lottor lo o moro thoorotlcjl concept. 
Slnco tooting for mony roooono ought to bo ^ 1 1 WW t od to o 
minimum, It lo importont to mcoouro ouch fotnolnt on« utt 
ouch tochnlquoo which porrHt mow I mum generel ll*t .on ocrooo 
domolno of con»ent. Domeln-eeloctlon "i 1 * 1 ** 
oooooood by coking oxporto to give JudfemenU on tho 
rolovonco of ooloctod domeh^. 

Pophm (1978) propoooo tho torm "doecflptlvo 
Ity" to Indlcoto tho repfeeentot I vendee of moeeurod eon- 
tont. In trodlilonol nomvrof oroncod tooting no quontltot- 
Ivo Indlcee ire uouolly given to doocfibo contont ro- 
prooontott nooo (cf. Totlo 1). In crIUrlon-rof oroncod 

tooting, ludgoo con bo uood to to whet 

•re congruont with tho toot opocif Icotlon. Hemiieton 
(1980) providoo •am uooful mothodo for doing thlo. ~n 
oomo orooo, whoro It lo poooiblo to epeclfy completely o 
pool of vohd toot Itwmn tho repreeentetl vonoto of Iteme 
con bo onourod by droning • rondom eemple from tno itom 
pool. This woo tho piocoduro odoptod whon tho preeent 
outhor etudlid otudonto* octivo ond peeelve vocobulory of 
Cnglloh In tho Tlnnloh eomprehenel ve ochool In lf7y. 

Hombloton (1980) Uooo tho torm "declelon volldlty* to 
rofor to tho doclolono mode on tho boolo of ocoroo. Pophom 
(1978) uooo tho torm "functlonel volldlty* in nwch tho 
oomo oonoo. DecSelon volldlty In cr I tor Ion-rot oroncod 
tooting lo ofton rolotod to otondord oottlng (minimum 
pooolng ocoroo). SSnco thot queetlon |» oornowhot beyond 
tho ecope of thlo popor It will not bo doolt w th further 
in thio context. A good roviow of declolon-conelotcncv lo 
in ^ubkovlek (1980). Hombloton ond Clgnor U?78) end 

»u„ ,.„„ ,.,.« .* ...... aVWIMe 
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for evaluating cr I t ar I on- re f er enced tests and test 
menua I s . 



6 Reliability as an Issue I n Cr • «>n-Refer enced 
Measurement 

Trad i t ional methods of est imat ing reliability in 
norm-referenced measurement are usually based on correlat- 
ional analyses where variance is a key concept. Since 
there may be relatively little variation in the scores of 
criterion-referenced tests, correl at I on-besed estimates 
may not be ideally suitable for the estimation of re- 
I 'abil ity. 

As Berk (1980) has noted there are at least three 
major conceptualization* of criterion-referenced test re- 
liability! 1) consistency of master y-non-master / decisiona 
acrosa repeated measures with one teat form or parallel 
test forma, 2) consistency of squared deviations of 
individual scores frun the cut-off scores across parallel 
or randomly parallel 'eat forma, 3) consistency of 
individual scores across parallel or randomly parallel 
test forms* 

Subkoviek (1980) givea a good survey of five methods 
of determining decision-consistency r e I i ab I li t y • Usua i I y 
only two statistics are used in this context I which 
indicates the proportion of individusla consistently 
clsssified as masters and non-masters across parallel test 
forms, and k , which estimates the proportion of individ- 
uals consistently classified beyond that expected by 
chance. Thus, P G estimates the overall consistency whoreaa 
* estimates consistency due to testing elone. The choica 
of the iriJix has to be based on whether one wants in 
estimate of overall consistency of decisions for whatever 
reason cr of the contribution of the test alone. In most 
cases, it Is probably advisable to report both estimates. 

Brennan (1980) rev i ews t he gene re I izebllity theory 
approach to reliability, which builds on the work by Cron- 
bach tnd his associates (1972). General i zab i 1 ity theory is 
based on the analysis of var i ance mode I and focuses on the 
estimation of various variance components in different 
types of test x items designs. General izebil i k y theory 
allows for the existence of many types and sources of 
error and it dots not require strictly psrallel trfsts for 
reliability estimation. Only randomly parallel tests are 
requ i r ed . 

As in the rase of the decision-consistency approach, 
there are two indices of reliability (or dependability)! 
♦(A) provides en est unite of the dependability of mastery- 
non-mastery decisions besed on the test ing procedure (X 
represents I he cur -off score), and + the "generel purpose" 
index that is i ^dependent of the cur-off score and which 
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ran bt itad to aa timet a Individual domain oeoroe (• major 
lnteraat In tht preeent writer 'e otudy of the »(n of 
otudonto' octlvo and peeelve vocabulary). |(\) lo rotated 
to tho rallablllty of cr I tor Ion-raff oreftced taet ayeroo ond 
i lo aeoocloted with tho rallablllty of oSSmS |g§f o 
ott > mo too* Tho formor Ino'catea how ctoeely tho rcoroa Tor 
any examinee con bo expected to agree, tho lotttr tho 
degree of ogroomont with ehonoo araamont removed/* thua 
} (A) choroctor I zoo tho dopondobl 1 1 ty of doclolo*i» r or 
eetlmetee, boood on tho tooting procedure* Ite magnitude 
depende, in port, on chonco ogroomont. tho ladOJif cH*r - 
ectorlzee tho contribution of tho tooting procedure to tho 
dopondobl 1 1 ty of dec le lone, over ond above what combo 01- 
poctod on tho boolo of chonco ogroomont (Brennan, 1910). 

Ao in tho cooo of tho doc la lon-conal a tency approach. 
It might bo uoaful to give both eetlmetee. Brannan (1910) 
•loo strongly rocomnondo thot vorlonco components too 
should olwoyo bo reported. 



7 Olocuoslon 

i , 

Cr I tor lon-rof oroncod fraoouromont and norm- ra far anted 
moaouromant ohara • number of feature*. Ae In ooverel 
othor fielde, for Inetenco, In curr'culum const ruct tan, 
now opproochss uouolly moon only now empheooe. At first 
thara la o tondancy to oxoggoroto dlfforoncoa. It lo 
pooolblo thot thlo Is Inevitable whan a new Idaa Is 
Int roducsd. Kor I Popper hoo suggootod that eartaln 
dooms t i may hova sn Important port to ploy In tho 
development of sclonco, bocouso giving up an Idaa tea soon 
may moor thot Itr mar I to ond wooknoooao ara not gl van a 
eufflclenl chonco of showing themeolvoa. A sclontlst 
should not bo too raady to adopt o now Idaa ar to abandon 
on old ono without persisting In soma seemingly dogmatic 
etonce for soma time for the ooko of argument. *Jo should 
know how to ploy tho believing ond doubting gomoo In s 
bsl onced way • 

Cr Iter lon-rof eranced meeeuroment shows some char* 
actorletlce of thla Initial dogmatism. At flret It wee 
cetegorlcel ly atated thot CUM doea not need ouch eoncopte 
ee Item end ecoro vorloncof thot empirical Item anetyeee 
srs not nssdsdf thot norm dote ehould not be* gathered* and 
thst content validity la the moot Important eepoct of CRM. 
It wae aoon odmltted, however, thot theeo claims were 
overeteted. Mom vsrlsnco uouolly occure *nd eOrvoo a uaa- 
ful purpoee In CRM tooting ee well ee In ..jrnvref oroncod 
tooting. Similarly, it wae conceded thot norm dote ere not 
amborreeelng for CRM. On the contrary, they add uaaful 
Information and can help to Interpret how "good" le "good 
enough". A poeterlorl empirical Item onelyeaa complement e 
priori judgemental ( ret I onel • log Ice I ) Itomonelyele ond 
hslp to do tact flowed Items. And, f I no 1 1 y, content 
validity la not the e 1 1 - Import ent conelderet Ion In CRM 
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Wh 1 1 • content represent at i venen is a necesssry charecter- 
i • t i c of CRM it does not guarantee thu validity of inter- 
pratationa baaed on CKT acorea. 

Cr I tar ion- referenced meeaitrement hea the apecial ad- 
vantage that it providea an exact description of a per- 
aon'a performance level in an entire domain and not only 
on the praaented itema. Severel requi rement a muat be ful- 
filled before auch an interpretation ia possible, f Irat, 
there hea to be e deteiled deecri^Mon of the meeaured 
domain. Second, there must be e deteiled deacription of 
the instrument, which includea the apec 1 f i cat i on of the 
etimulua and reaponae parte and of the acoring ayatem. 
Third, itema muat be genereted thet heve a high item- 
objective congruence end wh I ch ere a 1 ao a repr eaent at i ve 
rendom or it ret if led rendom anmple from the item pool . If 
CRM ia uaed for program eveluetion there muat alao be a 
repreaent et 1 ve aample of atudenta from the entire populat- 
ion. In the latter caae it ia adviaeble to us* matrix 
aa.np ling with aeverel per e 1 1 e 1 teat veraiona rotated in 
the c 1 aaa . 

One of the greeteat ettrecttona of CRM for the pre- 
eent writer ia ita empheaia on the concept ua 1 i /et i on of 
meeaured domaina. Thfc lenda aupport to hia peraonal 
claim, which goea beck aeverel yeera, thet one of the 
greatest obataclea for the development of teeching la the 
lack of theoret icel 1 y aound concept ue 1 i zet i ona of the 
unite jnd proceaaea in leerning e perticuler subject 
matter. He would, therefore, fully agree with the view 
recently put forwerd by Pophams 

When creeted oy inatruc Hon ally eatute developer a , a 
criteri on -referenced teat can ley out ao 1 uc i 4 1 y e 
aet of teecheble skills that the teat itself becomes 
e potent force for instructions! improvement. Inateed 
of being en efterthought for uae et the cloae of in- 
struction, a proper 1 y concept ue 1 1 zed criteri on -refer- 
enced teat can at imulete meeau rement -dr i ven inat ruct - 
ional enhen cement . Teat devel opera cen 1 i t erel 1 y 
create teat itema ao that they egree with one or more 
inat ruct I onal I y powerful explenetory conatructa which 
teacher a can then amp 1 oy dur i ng the i r I eaaone 
Thia aort of focuaed Inatructionel enterprise ie not 
teechj nq- to- 1 he- 1 eat In the neget I ve aenae the t one 
teechea Tower? e perticuler aet of teat itema. 
Rather, thia approech const Iturea l^icM ng- t o- the- 
aklll, a highly effective and thoroughly ~de7ens"rETe 
Tnsfruct ional atrategy" (Popham, 1981, pp. 106-107). 

Thus it might be thet "the tearing tail wegglng the 
teaching dog" may not be auch a problem or the emberraaa- 
rnent il ia often taken to be if the tell la fully compat- 
ible with rhe dog. The present writer's peraonal experi- 
ence with curriculum construction end evaluation, and with 
the in-service educetion of teachers in Finland auggeata 





thot thi moit effective end footoot wey to promote *oclt» 
•bio chengoo In toochlng lo to moke euro thot toot two, mm 
tooto dlepley tho character letlce of deelreble 
performance. Tooto oro tho moot concroto weyo of elgnollnf 
to toochoro end otudonto whet tho deolroblo oontont eft* 
(or™ of looming oro* 

Foouoina on tooting bo moro effects tMo *««u0» 
Ing on currlculo ond toochlng matoHoio eince teetlejej 000 
o moro llmltod ocopo ond It lo, therefore, peeelble to 
produco very corofully conotructod tooto that oro, In o 
oenoo, moduloo of toochlng. Such tooto ton iOroe eO 
oxemploo for proporlng unlto of toochlnf on* *•* 
dividual loooono. By concont rot ing on Importont eepewte wt 
tho eubjest mottor It lo pooolblo to produco ouch meOuIee 
which con oloo oorvo oo o otlmuluo for textbook 
While Indlvlduol unlto end me Jul ee we not eonotltulo on 
ontlro oyllebuo, thoy oro ueeful wtioloo oo ouch oM w 
oorvo oo ucoful model a. Proc:icol experience ohowe ™ot " 
U much moro difficult to oook to conceptually < ohtlrO 
curricula with olmllor rlfor ond It lo oloo o h tee* 
to produco o toxtbook pockogo with o olmllorly c ^letent 
approach. Thue tooting mey. Indeed, bo o oonolbl' 'tertlnf 
point ond leed to Improved currlculo ond textbooks. At tne 
vory leeet, tho potentiel contribution of work dono within 
tooting ond meeeuroment to curriculum doolgn ond instruct- 
ion should not bo Ignored. 

If wo continue to do oorlouo work on tooting, w# OOll 
movo from whet fine might deocrlbo eo tho modern "toot 
cult* moro ond moro towordo "toot culture". Toot ««|ture 
It cheroctorlxod by ooverol doolroble fOoturoo. 
thoro lo en oworenooo of the Importonce of knowing oOljr 0«o 
lo tooting In tho flrot piece. Second, thoro lo on ewere- 
n>'i of tho problomo of how vol Id Interpretet lont (con- 
clu. «cn w. mode on the beolo of obtained, more or 
I... rolloblo, ocoreo. Third, there lo on oworenooo +f 
problomo of tho gonorel Irebl I Ity of the reoulto to the 
whole content unlvoroe ond to the whole otudont P°P"J«^: 
Ion. Fourth, end flnolly, there lo en ebeenco of *>gme*lem 
end tebooo concerning toot typee. Toot culture lo mature 
when wo ore owero of every eopect of tooting ond evaluat- 
ion being riddled with probloma but wo ere only plooeod to 
have been able to reech ouch e level of owereneoe. 
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COHTCRTS LANGUAGE KWNUCtt 



LAWUAtt SKILLS 



or 

*cri»( tv 



Le**ls of language/ dts.rete ititi Sub-skills/ Integrative tests 



commcATiw siilis a 



USC Sf MS) 



STNTAI 



rrifraettc tests 



HTCHANlCAl 

StlllS 

• f p*,* 

in i «• 
Mil' 



Witch it 
dif ffrfnt ' 

W>l If h irf 
tdentliat ' 
Iwitate' 



» voiced, 

"first ' 



i <i ■ Short 
•Out '"to- 



sehmtics 



l »stenlng coapr* Speaking 
hens ton 



Reeding eoapre- Writing 
hens ton 



Recall 

conjugation, 
paradigm 

in accord- 
ant* with the 
•roe 1 ' 

»*C i tf * rulf 



give 

1 1 *<juw 
alfnt 



*uit»pir choic* >*jU»pir 

"atthm, 

•for,in waiton 



whet rm^*' 

j mg/ pi ur 1 1 7 

*it cat*' 

What dOH » 

••f^er to 7 

Which ttrvt- 
tur* fits 



What word 
c lass 1 
What t >h 

•wan mq 0 f 
1 1 ul inq 
fcOrd for** 
tiOfl i' 



Writ* down phone Imitate* 
Repeat 1 
Respond to 



nuwaers 
Spell nM«i 



Mitch tt dtfftr- Copy' 
ent> 

w*ich are the 



»**t wai laid 7 

What racti were 
stated 7 

Whet Opinions 
wtrf tiprrssed 7 



Convent tone! gesture* 
and faciei npresstont 
»n cower sat lone I 
interactivity 

Interpret' 

Use gestures . . f 



Convent lone i verbal 
expressions cuttoaery 
In vertel interaction 
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Act (speak/write) In a 
aanner ei pec ted in 
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>ta tp/preient/ 
inforn/f ipr»'t 



(9 sailing (tfipt wtre Dictation 



gO' n l to sa/U or a 

future with the thuaa 
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What was 

written' 

What racti/ 
opinion! 
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-< 
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'*oonst ratr. 



Try* raw* 

•Vltiplr C h 
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Question answer 

Error spotting 



What \trvr fUT 

would you usr 
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What is 'hr what 4oes the 
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verb rtc 
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word* 



Reading a loud 
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errata 



i g noveaent of hand 
with the pal* upwards 
(Ladles first" would 
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thmf as this point*) 
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Chatting 
Report Inaj 
Cloft 



Rnpont: and act 
craat tvely 
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craatwely 
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Under I in ing 
•a In points 
nahlnaj infer- 
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Correcting 
errors 
Crtttctf ing 

tait 



Reproduc t - 
ton 



(g sailing tn this 
contait I says to V 
"how about soap 
coffee?' 1 vat In 
and t ran sf eras 
malted request 
Into an offer 



Interview 
Conversation 
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narrating 
Free writing 
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